QTM 385 - Experimental Methods

Lecture 10 - Two-sided non-compliance

Danilo Freire
danilo.freire@emory.edu

Emory University

Hi, there!
Hope all is well! 😉

Brief recap 📚

Last class

  • One-sided non-compliance occurs when units in the treatment group fail to receive the intervention, while control group members remain unaffected
  • This scenario introduces two key groups:
    • Compliers: Subjects who accept treatment when assigned
    • Never-takers: Subjects who reject treatment regardless of assignment
  • The intent-to-treat (ITT) effect measures the impact of treatment assignment, while the complier average causal effect (CACE) estimates the true effect on those who actually received treatment
  • CACE can be calculated via: \(CACE = \frac{ITT_Y}{ITT_D}\)
  • Instrumental variables (IV) methods like two-stage least squares (2SLS) are preferred for estimation:
    • Regress treatment receipt on assignment (D ~ Z)
    • Use predicted treatment status to estimate outcome effects (Y ~ D_hat)
    • estimatr:::iv_robust(Y ~ D | Z, data = data)
  • Placebo designs help validate IV assumptions by testing compliers against non-treatment conditions
  • We should anticipate non-compliance through large sample sizes and robust experimental designs to maintain internal validity

Today’s plan 📋

Two-sided non-compliance

  • Two-sided non-compliance:
    • Treatment group: Some units don’t receive treatment
    • Control group: Some units access treatment externally
  • Four compliance types:
    • Compliers: Follow assigned treatment
    • Never-takers: Never receive treatment
    • Always-takers: Always seek treatment
    • Defiers: Do opposite of assignment
  • Non-ignorable selection and cross-contamination between arms
  • Monotonicity (no defiers)
  • Encouragement designs, double randomisation, and noncompliance-adjusted power calculations

Source: Alves (2022)

Two-sided non-compliance 🤔

Non-compliance: You already know it’s a problem

Now it will get worse 😂

  • Last class, we discussed one-sided non-compliance
  • We saw that ATE is not estimable in this scenario, as treatment and control groups are no longer comparable (selection bias risk)
  • We also learned how to estimate the CACE using IV methods, and it is the difference in observed treatment and control group outcomes by the proportion of subjects who are Compliers
  • Today, we will discuss two-sided non-compliance, which is even more complex
  • In this scenario, some subjects in the treatment group do not receive treatment, while some in the control group do
  • Underestimation of treatment effect (usually): If some in the treatment group don’t comply, and some in the control group get treatment, the difference between the groups in terms of actual treatment received becomes smaller
  • This can make it look like the treatment is less effective than it actually is. We “dilute” the treatment effect 🤓

Compliance types 📊

Four types of compliance

  • So far, we have discussed Compliers and Never-takers
  • In two-sided non-compliance, we also have Always-takers and Defiers
  • Always-takers: Subjects who always seek treatment, regardless of assignment
  • Defiers: Subjects who do the opposite of their assignment: Imagine stubborn teenagers! 😂
  • Many experiments face this issue, especially in social sciences
    • Encouragement designs: For example, students who receive private school vouchers but still attend public schools, but some students in the control group attend private schools even without vouchers
    • Natural experiments: Lottery defined who would be drafted to the Vietnam War, but some drafted soldiers avoided service, while some non-drafted soldiers volunteered
  • Fortunately, the estimation is similar to one-sided non-compliance, just with more assumptions

Four types of compliance

  • More formally, we have the following:

  • \(Z_i\) is the treatment assignment, \(D_i\) is the treatment receipt, and \(Y_i\) is the outcome

  • Compliers: \(D_i(1) = 1\) and \(D_i(0) = 0\). Similarly, \(D_i(1) \gt D_i(0)\)

  • Never-takers: \(D_i(1) = 0\) and \(D_i(0) = 0\)

  • Always-takers: \(D_i(1) = 1\) and \(D_i(0) = 1\)

  • Defiers: \(D_i(1) = 0\) and \(D_i(0) = 1\). These are usually rare, though

  • Connections between observed data and compliance types:

\(Z_i = 0\) \(Z_i = 1\)
\(D_i = 0\) Never-taker or Complier Never-taker or Defier
\(D_i = 1\) Always-taker or Defier Always-taker or Complier
  • Notice that treatment assignment has no effect on whether always-takers or never-takers are treated
    • Always-takers are treated regardless of assignment, while never-takers are never treated
  • Defiers and compliers, on the other hand, respond to treatment assignment, but in opposite ways
    • So the problem is that we can’t tell who is who, and this makes estimation difficult! 🤔

How to solve this? 🤔

Motivating example: Candidate debate study

  • Mullainathan et al (2010) designed a study to measure the impact of watching a political debate on voting intentions
  • Treatment group: Encouraged to watch debate
  • Control group: Encouraged to watch non-political programme
  • Treatment defined as self-reported debate viewing
  • Always-Takers: Watch debate regardless of encouragement
  • Never-Takers: Never watch debate, even if encouraged
  • Compliers: Watch debate only when encouraged
  • Defiers: Watch debate only when discouraged (watch non-political programme)
  • Compliance type is a fixed attribute in this design
    • If the design was different, compliance types could change!

Quantifying compliance

Estimating group sizes

  • Let \(\pi_{AT}\), \(\pi_{NT}\), \(\pi_{C}\), \(\pi_{D}\) denote proportions of Always-Takers, Never-Takers, Compliers, and Defiers

  • Formulas to estimate these proportions:

    • Always-Takers’ share (\(\pi_{AT}\)): \[ \pi_{AT} = \frac{1}{N} \sum_{i=1}^{N} d_i(1)d_i(0) \]
    • Never-Takers’ share (\(\pi_{NT}\)): \[ \pi_{NT} = \frac{1}{N} \sum_{i=1}^{N} (1-d_i(1))(1-d_i(0)) \]
    • Compliers’ share (\(\pi_{C}\)): \[ \pi_{C} = \frac{1}{N} \sum_{i=1}^{N} d_i(1)z_i(1-d_i(0)) \]
    • Defiers’ share (\(\pi_{D}\)): \[ \pi_{D} = 1 - \pi_{AT} - \pi_{NT} - \pi_{C} \]

Quantifying compliance

Numbers from the debate study

  • Under random assignment, the assigned treatment group has the same expected shares of Always-Takers, Never-Takers, Compliers, and Defiers as the assigned control group
    • Right? Why or why not?
  • In the control group, the untreated subjects are either Never-Takers or Compliers
    • The study of the New York City mayoral debates found that 84% of the control group reported not watching the debate, so \(\hat{\pi}_{NT} + \hat{\pi}_{C} = 0.84\)
    • Subjects in the control group who watched the debate are either Always-Takers or Defiers, and \(\hat{\pi}_{AT} + \hat{\pi}_{D} = = 0.16\)
  • In the treatment group, 37% of the subjects reported watching the debate
    • These subjects must be either Always-Takers or Compliers, so \(\hat{\pi}_{AT} + \hat{\pi}_{C} = 0.37\)
    • The remaining 63% are either Never-Takers or Defiers, so \(\hat{\pi}_{NT} + \hat{\pi}_{D} = 0.63\)
  • But we don’t know how many people are in each group, as mentioned before
  • We can estimate these proportions with a trick…

Monotonicity 📏

Monotonicity assumption

No defiers allowed! 😂

  • Monotonicity is a key assumption in two-sided non-compliance
  • It states that no subject is a Defier: No one does the opposite of their assignment
  • This is a [strong assumption too], as it is possible that someone does the opposite of what they were told
  • But if we assume that is the case, all problems are solved! 😂
  • In our previous example:
    • If we assume that no one watched the debate when they were told not to, we can estimate the proportions of each group

Monotonicity assumption

Simplifying the Model

  • Assume no Defiers (\(\pi_{D} = 0\)) to simplify estimation
  • With \(\pi_{D} = 0\), we can estimate other proportions:
    • Always-Takers (\(\hat{\pi}_{AT}\)):
    • \(\hat{\pi}_{AT} + \hat{\pi}_{D} = 0.16 \implies \hat{\pi}_{AT} = 0.16\)
    • Never-Takers (\(\hat{\pi}_{NT}\)):
    • \(\hat{\pi}_{NT} + \hat{\pi}_{D} = 0.63 \implies \hat{\pi}_{NT} = 0.63\)
    • Compliers (\(\hat{\pi}_{C}\)):
    • \(\hat{\pi}_{AT} + \hat{\pi}_{C} = 0.37 \implies \hat{\pi}_{C} = 0.37 - \hat{\pi}_{AT} = 0.37 - 0.16 = 0.21\)
    • Alternatively, using:
    • \(\hat{\pi}_{NT} + \hat{\pi}_{C} = 0.84 \implies \hat{\pi}_{C} = 0.84 - \hat{\pi}_{NT} = 0.84 - 0.63 = 0.21\)
  • Both formulas yield approximately \(\hat{\pi}_{C} \approx 0.2\)

Now the easy part, CACE estimation 🤓

Estimating CACE

Now we know what to do! 😂

  • Once we eliminated Defiers, we can estimate the CACE using the same formula as before
  • The CACE is the difference in observed treatment and control group outcomes by the proportion of subjects who are Compliers
  • Although two-sided noncompliance introduces the possibility that some subjects are Always-Takers, they pose no identification problems
  • Always-Takers have no effect on the \(ITT\), and the share of Always-Takers is differenced away when we calculate the \(ITT_D\)
  • Why so? Because they are treated regardless of assignment, so they don’t affect the treatment effect!

ITT decomposition with no defiers

  • Intent-to-treat effect on outcome (\(ITT_Y\)) can be decomposed:

    \[ ITT_Y = ITT_{Y,co} \pi_{co} + \underbrace{ITT_{Y,at} \pi_{at}}_{=0} + \underbrace{ITT_{Y,nt} \pi_{nt}}_{=0} + \underbrace{ITT_{Y,de} \pi_{de}}_{=0 \text{ (mono)}} \]

  • Under Exclusion Restriction (ER) and Monotonicity (mono) assumptions, ITT simplifies to:

    \[ ITT_Y = ITT_{co} \pi_{co} \]

  • Same identification result:

    \[ \tau_{LATE} = \frac{ITT_Y}{ITT_D} \]